https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> how to write multi line conditional s...
# ask-marvin
o
@Marvin how to write multi line conditional script logic in prefect yaml deployment prefect.deployments.steps.run_shell_script section.
m
✅ thought for 85.4 seconds
You can write multi-line shell logic in a deployment step by using a YAML block scalar and the 
prefect.deployments.steps.utility.run_shell_script
step. A few tips: - Use 
script: |
for multi-line content. - Use 
env:
to set environment variables for the script. - Set 
expand_env_vars: true
if you want 
$VAR
or 
${VAR}
in your script to be expanded from the environment. - Keep your shell syntax POSIX-compatible (
sh
) unless you’re sure of the shell; simple `if`/`else` with 
[ ]
works well. - The correct step path in 3.x is 
prefect.deployments.steps.utility.run_shell_script
. Example with conditional logic:
Copy code
build:
  - prefect.deployments.steps.utility.run_shell_script:
      id: prechecks
      directory: .
      script: |
        set -euo pipefail

        echo "Starting prechecks for ${PREFECT_DEPLOYMENT_NAME:-unknown}"

        if [ "${RUN_DB_MIGRATIONS:-false}" = "true" ]; then
          echo "Running database migrations..."
          alembic upgrade head
        else
          echo "Skipping database migrations."
        fi

        if [ -n "${FEATURE_FLAG:-}" ]; then
          echo "Feature flag is set to: ${FEATURE_FLAG}"
        else
          echo "Feature flag not set."
        fi

        echo "Done."
      env:
        RUN_DB_MIGRATIONS: "true"
        FEATURE_FLAG: "beta"
      expand_env_vars: true
      stream_output: true
Alternative using a folded scalar (
>-
) if you prefer a single logical line (YAML will fold newlines to spaces; generally 
|
is safer for scripts):
Copy code
build:
  - prefect.deployments.steps.utility.run_shell_script:
      script: >-
        set -e;
        if [ "${ENVIRONMENT:-dev}" = "prod" ]; then
          echo "Doing prod setup";
        else
          echo "Doing non-prod setup";
        fi
      env:
        ENVIRONMENT: "prod"
      expand_env_vars: true
Docs: - run_shell_script API: prefect.deployments.steps.utility.run_shell_script If you’re hitting a YAML parsing or quoting issue, paste your current 
prefect.yaml
snippet and I’ll help fix the indentation/quoting.
o
@Marvin **- prefect.deployments.steps.run_shell_script: id: get-branch-name script: | if [ -n "$CI_COMMIT_REF_NAME" ]; then BRANCH_NAME="$CI_COMMIT_REF_NAME" else BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD) fi # Clean the branch name for a valid Docker tag # Replace '/' with '-', convert to lowercase, and remove non-allowed chars CLEAN_BRANCH_NAME=$(echo "$BRANCH_NAME" | tr '/' '-' | tr '[upper]' '[lower]' | sed 's/[^a-z0-9._-]/-/g' | sed 's/^-*//' | sed 's/-*$//') echo "$CLEAN_BRANCH_NAME" stream_output: false expand_env_vars: true** Whats wrong with this shell script
m
✅ thought for 102.8 seconds
A few things to fix/consider: 1) Use the correct step path in Prefect 3.x - It should be 
prefect.deployments.steps.utility.run_shell_script
, not 
prefect.deployments.steps.run_shell_script
. 2) Avoid a trailing newline in stdout - If you plan to use this step’s output as a Docker tag (or similar), 
echo
will include a newline. Prefer: - 
printf '%s' "$CLEAN_BRANCH_NAME"
(portable) instead of 
echo
. 3) Ensure you’re in a git repo (or set directory) - 
git rev-parse --abbrev-ref HEAD
will fail if the working directory isn’t a Git repo. Either: - Run a prior pull/clone step, or - Set 
directory:
to the repo path. 4) 
expand_env_vars
is optional here - The shell will expand 
$CI_COMMIT_REF_NAME
already. You can keep 
expand_env_vars: true
, but it’s not required. Corrected example:
Copy code
- prefect.deployments.steps.utility.run_shell_script:
    id: get-branch-name
    script: |
      if [ -n "$CI_COMMIT_REF_NAME" ]; then
        BRANCH_NAME="$CI_COMMIT_REF_NAME"
      else
        BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD)
      fi

      # Clean the branch name for a valid Docker tag
      CLEAN_BRANCH_NAME=$(echo "$BRANCH_NAME" \
        | tr '/' '-' \
        | tr '[:upper:]' '[:lower:]' \
        | sed 's/[^a-z0-9._-]/-/g' \
        | sed 's/^-*//' \
        | sed 's/-*$//')

      # Print without a trailing newline so consumers (e.g., Docker tag) don't break
      printf '%s' "$CLEAN_BRANCH_NAME"
    stream_output: false
    expand_env_vars: false
    # directory: /path/to/git/repo   # uncomment if needed
Tip for using the result later in your prefect.yaml: - Reference it as 
{{ get-branch-name.stdout }}
(note: if you use 
echo
, you’ll get a trailing newline). Docs: - run_shell_script API: prefect.deployments.steps.utility.run_shell_script - YAML/templating basics: prefect.yaml how-to If you’re seeing a specific error message, share it and I’ll pinpoint the exact cause.
o
> Running run_shell_script step... Traceback (most recent call last): File "/home/omar/23-0325_senec_monitoring/.venv/lib/python3.13/site-packages/prefect/deployments/steps/core.py", line 163, in run_steps step_output = await run_step(step, upstream_outputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/omar/23-0325_senec_monitoring/.venv/lib/python3.13/site-packages/prefect/deployments/steps/core.py", line 134, in run_step result = await from_async.call_soon_in_new_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Call.new(step_func, **inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ).aresult() ^^^^^^^^^^^ File "/home/omar/23-0325_senec_monitoring/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 338, in aresult return await asyncio.wrap_future(self.future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/omar/23-0325_senec_monitoring/.venv/lib/python3.13/site-packages/prefect/_internal/concurrency/calls.py", line 402, in _run_async result = await coro ^^^^^^^^^^ File "/home/omar/23-0325_senec_monitoring/.venv/lib/python3.13/site-packages/prefect/deployments/steps/utility.py", line 176, in run_shell_script async with open_process( ~~~~~~~~~~~~^ split_command, ^^^^^^^^^^^^^^ ...3 lines... env=current_env, ^^^^^^^^^^^^^^^^ ) as process: ^ File "/usr/lib/python3.13/contextlib.py", line 214, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/omar/23-0325_senec_monitoring/.venv/lib/python3.13/site-packages/prefect/utilities/processutils.py", line 212, in open_process process = await anyio.open_process(command, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/omar/23-0325_senec_monitoring/.venv/lib/python3.13/site-packages/anyio/_core/_subprocesses.py", line 190, in open_process return await get_async_backend().open_process( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...11 lines... ) ^ File "/home/omar/23-0325_senec_monitoring/.venv/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 2567, in open_process process = await asyncio.create_subprocess_exec( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...5 lines... ) ^ File "/usr/lib/python3.13/asyncio/subprocess.py", line 224, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...3 lines... stderr=stderr, **kwds) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/asyncio/base_events.py", line 1794, in subprocess_exec transport = await self._make_subprocess_transport( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ protocol, popen_args, False, stdin, stdout, stderr, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ bufsize, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/asyncio/unix_events.py", line 213, in _make_subprocess_transport transp = _UnixSubprocessTransport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, waiter=waiter, extra=extra, **kwargs) File "/usr/lib/python3.13/asyncio/base_subprocess.py", line 39, in init self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ stderr=stderr, bufsize=bufsize, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/asyncio/unix_events.py", line 855, in _start self._proc = subprocess.Popen( ~~~~~~~~~~~~~~~~^ args, shell=shell, stdin=stdin, stdout=stdout, stderr=stderr, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ universal_newlines=False, bufsize=bufsize, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/subprocess.py", line 1039, in init self._execute_child(args, executable, preexec_fn, close_fds, ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pass_fds, cwd, env, ^^^^^^^^^^^^^^^^^^^ ...5 lines... gid, gids, uid, umask, ^^^^^^^^^^^^^^^^^^^^^^ start_new_session, process_group) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/subprocess.py", line 1972, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'if' @Marvin
m
✅ thought for 84.3 seconds
The traceback tells us exactly what’s wrong: - FileNotFoundError: No such file or directory: 'if' Prefect’s run_shell_script executes the string by splitting it into tokens and running the first token as the executable (no implicit shell). Since your script starts with 
if
, Prefect tries to execute a program named 
if
, which fails. Fix: invoke a shell explicitly (bash or sh) and pass your multi-line script to it. Also use the correct step path in 3.x. Example:
Copy code
- prefect.deployments.steps.utility.run_shell_script:
    id: get-branch-name
    script: |
      bash -lc "
        set -euo pipefail

        if [ -n \"${CI_COMMIT_REF_NAME:-}\" ]; then
          BRANCH_NAME=\"${CI_COMMIT_REF_NAME}\"
        else
          BRANCH_NAME=\$(git rev-parse --abbrev-ref HEAD)
        fi

        CLEAN_BRANCH_NAME=\$(echo \"\$BRANCH_NAME\" \
          | tr '/' '-' \
          | tr '[:upper:]' '[:lower:]' \
          | sed 's/[^a-z0-9._-]/-/g' \
          | sed 's/^-*//' \
          | sed 's/-*$//')

        # Print without a trailing newline
        printf '%s' \"\$CLEAN_BRANCH_NAME\"
      "
    stream_output: false
    expand_env_vars: false
    # directory: /path/to/git/repo  # set if needed for git commands
Notes: - Using 
bash -lc "<script>"
ensures your multi-line conditional runs in a shell. - Double quotes around the script let environment variables expand; inner double quotes are escaped. - You can swap 
bash
for 
sh
if bash isn’t available. - 
expand_env_vars
isn’t required here—the shell expands 
$VAR
. Docs: prefect.deployments.steps.utility.run_shell_script
Open in Slack
PreviousNext
Powered by